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(54) Robot system having image processing function 

(57) A robot system has an image processing func- 
tion capable of detecting position and/or posture of in- 
dividual workpieces randomly arranged in a pile to de- 
termine posture, or posture and position of a robot op- 
eration suitable for the detected position and/or posture 
of the workpiece. Reference models are created from 
two-dimensional images of a reference workpeace cap- 
tured in a plurality of directions by a first visual sensor 
and stored. Also, the relative positions/postures of the 
first visual sensor with respect to the workpiece at the 
respective image capturing, and relative fios it ion/pos- 
ture of a second visual sensor to be situated with respect 
to the workpiece are stored. Matching probessing be- 
tween an image of a pile of workpieces captured by the 
camera and the reference models are performed and an 
image of a workpiece matched with one reference mod- 
el is selected. A three-dimensional position/posture of 
the workpiece is. determined from the image of the se- . 
lected workpiece, the selected reference model and po- 
silion/posture information associated with the reference 
model. The position/posture of the second visual sensor 
to be situated for measurement is determined based on 
the determined position/posture of the workpiece and 
the stored relative position/posture of the second visual 
sensor, and precise position/posture of the workpiece is 
measured by the second visual sensor at the deter- 
mined position/posture of the second visual sensor. A 
picking-up operation for picking out a respective work- 
piece from a randomly arranged pile can be performed 
by a robot, based on the measuring results of the second 
visual sensor. 
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[0001] The present invention relates toa robot sy=tem 
having an image processing function fordetecun fhte 
d-mensjonal position and posture (orientation) of an ob 
ject. and ,n part-cularto a robot system having an fmaoe 
pro.ess.ng function sui.abie for a bin-pickingopeSn 
fartokng out workpiece one by one from a random^ 
arranged pile of workpieces. ' - 
[0002] The operation of taking out an individual work- 
Z laarln f Pi ' S ° f WO *P'eces or 

of a ored? 00 ° f ^ 0rkpieCeS C ° ntained in a ««teiner 
of a predetermined size, which workpieces have iden.i- 

Oos,urL PeS H a h ffer6nt th ^<™™°™< Positions/ 
postures, has been performed manually In storino 
workp, on a pa| , et of J n s ^ -J 

determined pos.tion in a machine or a device using a 
dedicated) robot, since it has been impossible ,2 
. y take out individual workpieces one by one from the 

ob^hrbT 96 " Pi ' e ° f WOrkPiSCeS b " ,he ded -'^ 
robot rt hdS been neC essary to rearrange the workpiec- 
es in advance so as to be picked out by the robot In this 
rearrangement operation, i, has been necessary to tike 

' r' h VidUal WOfkpieCe fr ° m ,he pi,e manually. 
2, L hG reaSO " Wh * individua < workpieces having 

n t d,fferent thre ^rnensiona. Posi 

t-ons/postures cannot be picked out by a robot from a 

randomfy arranged pile of workpieces or an aggregXn 

of workpieces contained in a.container is thaUhe pS 

tion/posture of indKridua. workpieces in the pile on 

c, 9 nn e fK tl0n , ° ann0t bS reco 9 nised so that a robot hand 
cannot be placed to a suitable position/posture at which 

moai . ^ h °' d ' he individual wo ^piece 
[0004] An object of the present invention is to provide 

pable of S d S Jt em 3Vin9 ^ ima9e Pr ° Cessin 9 ^'on ca 
pable o determ.nmg an orientation and/or a position of 
a robot for performing an operation on an objeci b™ 
c.sely detecting three-dimensional posrtion andWu* 
of .ndiv.dual objects in a randomly arranged pile o an 

SE^T 3 C ° ntainer ' n 3 P-determined regon 
which have .dentica. shapes and different three-dimen 
sional positions/postures. 

[0005] A robot system of the present invention having 
an .mage processing function comprises a robot a first 
^capturing device, a memory La processor The 
memory stores reference models created according to 

caTrif h °' 9 refSfenCe ^ CaptUf6d * 
captunng device ,n a plurality of directions, and stores 

."formation of the capturing directions to be especS 

assoca ed with the reference models and informS 

of onen.at.on of the robot operation with respect to the 

ora^oJ^ n eferenCe *** iS the ^ ^detection 
or an object having a shape identical to that of the object 
for detection. The processor performs matching 
processing on image data containing an image for the 
object of detection captured by the firs, image captur ng 
dev.ee w , th said reference models to selecta n image of 
an object matched with one of the reference mooels 



and determines orientation, or orientation and position 
of an operation to be performed by the robot based on 
the selected image of the object, said one reference 
model and information about the capturing direction and 
information about the orientation of the robot operation 
with respect to the object associated with said one ref- 
erence model. 

[0006] With a robot system of the present invention 
a robot operation can be performed on an individual ob- 
ject m a p.le or on an aggregation of plural kinds of ob- 
jects. In th.s case, image data of plural kinds of reference 
objects are captured by the firs, image data capturing 
dev.ee lo create the reference models based on the cap 
tured ,mage data, and relevant information is addition- 
ally stored to be associated with each reference model 
Each of the reference objects is the object of each kind 
of operation or an object having a shape identical to that 
of the object of operation of each kind. The processor 
determines orientation, or orientation and position, of 
the robot operation based on ihe image of the object 
and one reference model selected in the matching 
processing, and the information of the kind associated 
w.th sa.d one. reference model and the information of 
the onentation of the robot operat.on with respect ,o the 
object associated with said ono reference model 
[0007] The .mage capturing device may be a camera 
or capturing two-dimensional images and in this case 
he image data of the reference model are captured by 
the camera from a predetermined distance 
[0008] The robot may situate the second image data 
captunng device to have the determined orientation or 
to have the determined orientation and the determined 
posihon with respect to the object, and the processor 
may process second image data captured by the sec- 
ond .mage capturing device to detect position and/or 
posture of the object with respect to the second image 
data capturing device. 

[0009] The robot may also situate the second image 
40 £ n a C ? T 9 d6ViCe IO have * e determined orienfa- 
.on or to have the determined orientation and the de- 
termined position with respect to the object, so that the 
second .rnage data capturing device is directed to a 
characterising portion of the object, and the processor 
45 T V 1 J hww *™««tonal posit.on and/or posture of 
the object based on three-dimensional position of said 
characterising portion obtained by the second image 
capturing device. y 
[0010] The first image data capturing device can be 

so mntf SeC ° nd image data ca P*™ g device. 

lopil] The second image capturing device may com- 
prise a three-dimensional visual sensor of spot-liqht 
scanning type capable of measuring distance between 
he sensor and an object, or may comprise a structured- 
hght unit for .rradiating a structured light on an object 

32 hT'k." 19 a " ima9e ° f thS ° bject includin 9 the irra- 
diated light on the object. 

[0012] The robot operation may be an operation for 
pickmg up at least one object from a plurality of objects 
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whose positions overlap. 

[0013] For a better understanding of the invention, 
and to show how the same may be carried into effect, 
reference will now be made, by way of example, to the 
accompanying drawings, in which:- 

FIG. 1 is a diagram for showing a picking-up oper- 
ation by a robot to take out an individual workpiece 
from a pile of workpieces using an image process- 
ing apparatus according to an embodiment of the 
present invention; 

FIGS. 2a-2d show an example of reference models; 
FIG. 3 is a block diagram of a principal part of a robot 
controller; 

FIG. 4 is a block diagram of the image processing 
apparatus according to an embodiment of the 
present invention; 

FIG. 5 -is a flowchart of the processing for creating 
reference models; 

FIG. 6 is a flowchart of the processing for Lhe .pick- 
ing-up operation; 

FIG. 7 is a table showing relative positions/postures 
of the workpiece relative to the first visual senor and 
the positions/postures of the second visual sensor 
to be situated relative to the workpiece in the case 
of four reference models; 

FIG. 8 is a diagram showing an example of scanning 
motion of a visual sensor capable of obtaining dis- 
tance data; 

FIG. 9 is a diagram of the two-dimensional arrange- 
ment data containing distance data as image data 
obtained by the visual sensor; 
FIG. 10 is a flowchart of processing for obtaining 
the two-dimensional arrangement data. 

[0014] A robot system having an image processing 
function according to an embodiment of the present in- 
vention will be described. As shown in FIG. 1,-a two- 
dimensional visual sensor 21 such as a CCD camera, 
is attached to a wrist of a robot RB as an image capturing 
device. In this embodiment, an image of a pile of work- 
pieces, which are objects for detection having identical 
shapes and randomly arranged as shown in FIG. 1, is 
captured by the two dimensional visual sensor 21, and 
orientation, or orientation. and position, of an operation 
to be performed by the robot RB is determined based 
on the captured image. Further, rough position and pos- 
ture of the individual workpieces are detected to deter- 
mine the orientation of the robot based on the captured 
image and then precise position and posture of the 
workpiece are detected by the two-dimensional visual 
sensor 21 or a three-dimensional visual sensor 22. 
[001 5] For this purpose, images of a reference object, 
which is one of workpieces W subjected to a picking-up 
operation or an object having a shape identical to that 
of the workpiece W are captured in different directions 
by the image capturing device and reference models are 
created from the image data obtained by the image cap- 



turing and stored in advance. Matching'processing be- 
tween the image data obtained by capturing the image 
of the pile of workpieces and the reference models is 
executed to select an image of one workpiece matched 

5 with one of reference models, and the position/posture 
of the selected workpiece is determined based on the 
selected image of the workpiece in the image field of 
view, the selected one of reference modes and the po- 
sition/posture information associated with the selected 

to one of the reference models. 

[0016] FIG. 3 is a block diagram showing a principal 
part of a robot controller 1 0 for use in an embodiment of 
the present invention. A main processor 1 , a memory 2 
including a RAM, a ROM and a nonvolatile memory 

75 (such as an EEPROM), an interface 3 for a teaching op- 
erating panel, an interface 6 for external devices, an in- 
terface 7 for an image processing apparatus and a servo 
control section 5 are connected to a bus 8. A teaching 
operating panel 4 is connected to the interface 3. 

20 [0017] A system program for supporting basic func- 
tions of the robot RB and robot controller 10 are stored 
in the ROM of the memory 2. Robot operation programs 
and their related determined data which are taught in 
accordance with various operations are stored in the 

2S nonvolatile memory of the memory 2. The RAM of the 
memory 2 is used for temporary storage of data for var- 
ious arithmetical operations performed by the processor 
1. 

[0018] The servo control section 5 comprises servo 
30 controllers Sal to San (n: sum of the number of all the 
axes of the robot including additional movable axes of 
a tool attached to the wrist of the robot), each composed 
of a processor, a ROM, a RAM, etc. Each servo control- 
ler rjerforms position/velocity loop control and also cur- 
35 rent loop control for its associated servomotor for. driving 
the axis, to function as a so-called digital servo controller 
for performing loop control of position, velocity and cur- 
rent by software. Each servomotor Ml-Mn for driving 
each axis is drivingly controlled according to outputs of 
•*o the associated servo controller 5al-5an through the as- 
sociated servo amplifier 5bl-5bn. Though not shown in 
FIG. 3, a position/velocity detector is attached to each 
servomotor Ml-Mn, and the position and velocity of each 
servomotor detected by the associated position/velocity 
detector is fed back to the associated servo controller 
5al-5an. To the input/output interface 6 are connected 
sensors of the robot, and actuators and sensors of pe- 
ripheral devices. 

[0019] FIG 4 is a block diagram of the image process- 
50 ing apparatus 30 connected to an interface 7 of the robot 
controller 10. The image processing apparatus 30 com- 
prises a processor 31 to which are connected through 
a bus 42 a ROM 32 for storing a system program to be 
executed by the processor 31 , an image processor 33, 
55 a camera interface 34 connected to a CCD camera 21 
which is a first visual sensor serving as a first image data 
capturing device, a three-dimensional sensor interface 
35 connected to a three-dimensional sensor 22 as a 
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second visual sensor as a second image data capturing 
device, a MDI 36 with a display such as a CRT or a liquid 
crystal display for inputting and cutputting various com- 
mands and data a frame memory 37, a nonvolatile 
memory 38. a RAM 39 for temporary storage of data, a 
communication interface 40 for the robot controller and 
a console interface 41 for a console 43. An image cap- 
tured by the CCD camera 21 is converted into a light 
and shade grey scale image and stored in the frame 
memory 37. The image processor 33 performs image 
processing of images stored in the frame memory 37 on 
demand by the processor 31 so as to recognise an ob- 
ject. The architecture and function of the image process- 
ing apparatus 30 itself is in no way different from the 
conventional image processing apparatus. The image 
processing apparatus 30 of the present invention is dif- 
ferent from the conventional one in that reference mod- 
els as described later are stored in the nonvolatile mem- 
ory 38 and pattern matching processing is performed on 
an image of a pile of workpieces W captured by the firsl 
visual sensor 21 using the reference models to obtain 
position and posture of a workpiece W and thus orien- 
tation or orientation and position, of an operation to be 
performed by the robot RB. Further, the image process- 
ing apparatus of the present invention differs from the 
conventional one in that approach orientation/position 
for more precise detection of the position/posture of the 
workpiece W by the second visual sensor of the three- 
dimensional visual sensor 22 is determined based on 
the detected position/posture of the workpiece W using 
the first visual sensor 21 . 

[0020] The CCD camera for obtaining two-dimension- 
al images data is well known in the art and thus detailed 
explanation thereof is omitted here. The three<Jimen- 
sional visual sensor 22 for detecting a three-dimension- 
al position of an object by irradiating structured light (slit 
beam) on the object is known and used in various fields. 
This type of sensor may be used as the thpe dimen- 
sional visual sensor 22. Further, a three-dimensional 
visual sensor of a spot-light scanning type as disclosed 
in Japanese Patent Publication No. 7-2701 37 may be 
used as the three-dimensional visual sensor 22, and the 
summary of such three-dimensional visual sensor is de- 
scribed below. 

[0021] This visual sensor detects a three-dimensional 
position of an object by irradiating a light beam to form 
a light spot on the object for scanning the object in two 
different directions (X direction and Y direction) and by 
detecting the light reflected on the object by a position 
sensitive detector (PSD). The three dimensional posi- 
tion of the object is measured by a calculation using the 
respective inclination angles 9x,9y of mirrors for scan- 
ning and incident positions of the reflected light beam 
on the PSD. 

[0022] Referring to FIGS. 8-10. a method of obtaining 
two-dimensional arrangement data including distance 
data using the three-dimensional visual sensor will be 
explained briefly. 



[0023] The scanning range (measuring range) on an 
object is set in advance, and an inclination angle 8x.9y 
of the mirrors is controlled discretely. As shown in FIG. 
7 : the scanning is performed from a point (1 , 1 ) to a point 
s (1 , n), from a point (2,1) to a point (2. n), .... from a point 
(m, 1 ) to a point (m, n) on the X-Y plane within the scan- 
ning range, to measure the three-dimensional positions 
of each reflected point on the object. Also, a distance 2 
(i, j) between the sensor and the reflection point (i, j) on 
io the object is obtained and stored in the RAM38 of the 
image processing apparatus 30. Thus, the image data 
is obtained as two-dimensional arrangement data in- 
cluding the distance data 2 (i, j) between the sensor and 
the reflection point on the object, as shown in FIG. 9. 
is [0024] FIG. 10 is a flowchart of processing to be exe- 
cuted by the processor 31 of the image processing ap- 
paratus 30 for obtaining the image data. 
[0025] First, indexes i and j are respectively set to "1 " 
(Step 300) and the inclination angle (Bx,ey) of the mir- 
20 rors is set lo (x1 , yl) lo direct to the start point (1,1) and 
an irradiation command with the inclination angle is sent 
to the sensor 20 (Steps 301-303). The sensor irradiates 
a light beam with the mirrors set at the inclination angle. 
The signal representing the image captured by the PSD 
25 is sent to the image processing apparatus 30. The proc- 
essor 31 of the image processing apparatus 30 calcu- 
lates the position of the reflection point on the object 
from the. signal from the PSD and the inclination angle 
(Bx.Gy) of the mirrors to obtain the distance 2 (i, j) be- 
30 tween the sensor and the position of the reflection point 
on the object. This value 2 (i, j) is stored in the RAM 38 
as the two-dimensional arrangement data [i, j] (Step 
304,^305). The calculation for obtaining the position of 
the reflection point and the distance 2 (i, j) may be per- 
35 formed by the sensor 20. 

[0026] Then, the index i is incrementally increased by 
"1 " and the inclination angle Dx of the mirror for X-axis 
direction scanning is increased by the predetermined 
amount Ax (Step 306, 307). It is determined whether or 
4 o not the index i exceeds the set value n (Step 308). If the 
index i does not exceed the set value n, the procedure 
returns to Step 303 and the processing from Step 303 
to Step 308 is executed to obtain the distance 2 (i. j) of 
the next point. Subsequently, the processing of Steps 
45 303-308 is repeatedly executed until the index i exceeds 
the set value n to obtain and store the distance 2 (i, j) 
of the respective points (1, 1) to (1, n) shown in FIG. 8. 
[0027] If it is determined that the index i exceeds the 
set value n in Step 308, the index i is set to "1 " and the 
50 index j is incrementally increased by T to increase the 
inclination angle 8y of the mirror for Y-axis direction 
scanning (Steps 309-31 1 ). Then, it is determined wheth- 
er or not the index j exceeds the set value m (Step 31 2) 
and if the index j does not exceed the set value m, the 
55 procedure returns to Step 302 to repeatedly execute the 
processing of Step 302 and the subsequent Steps. 
[0028] Thus, the processing from Step 302 to Step 
31 2 is repeatedly executed until the index j exceeds the 
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set value m. If the index j exceeds the set value m, the 
points in the measurement range (scanning range) 
shown in FIG. 8 have been measured entirely, the dis- 
tance data 2 (1, 1) - Z (nr n) as two-dimensional ar- 
rangement data are stored in the RAM 39 and the image 5 
data obtaining processing is terminated. A part of the 
image data of two-dimensional arrangements or a plu- 
rality of distance data can be obtained by appropriately 
omitting the measurement of the distance for the index i. 
[0029] The foregoing is a description on the process- 
ing for obtaining two-dimensional arrangement data as 
image data using the visual sensor capable of measur- 
ing the distance. Using the two-dimensional arrange- 
ment data obtained in this way as image data, creation 
of reference models and detection of position and pos- 
ture (orientation) of an object can be performed. 
[0030] In order to simplify the explanation, the follow- 
ing description will'be made assuming that a CCD cam- 
era is used as the two-dimensional visual sensor 21 and 
the two-dimensional image data obtained by capturing 
image of the object by the CCD camera is used. 
[0031] Processing for creating reference models will 
be explained referring to FIGS. 2a-2d and FIG. 5. FIG. 
5 is a flowchart showing processing for teaching refer- 
ence models to the image processing apparatus 30 ac- 
cording to the present invention. 

[0032] One reference workpiece (one of the workpiec- 
es Was objects of robot operation or a workpiece having 
a three-dimensional shape identical to that of the work- 
piece W) is arranged at a predetermined position with 
predetermined posture for creating reference models. A 
first (0-th) position and orientation (posture) of the CCD 
camera 21 for capturing the image of the object is set, 
and an axis of rotation and rotation angles with respect 
to the first (0-th) position and orientation (posture) are 
set in order to determine the subsequent positions and 
orientations (postures) of the camera 21 from the teach- 
ing operation panel 4 of the robot controller 10. In addi- 
tion, the number of the positions and orientations (pos- 
tures) of the workpiece at which the camera 21 captures 
the image of the object are set. 

[0033] As shown in FIGS. 2a to 2d, in this example, 
images of the reference workpiece are captured from 
four different directions and reference models are cre- 
ated based on the four image data. As shown in FIG. 
2a, an image of the reference workpiece is captured 
from the direction of a Z-axis of a world coordinate sys- 
tem at 0-th position/orientation (posture) to create 0-th 
reference model. For setting the subsequent positions/ 
orientations (postures), an axis perpendicular to an op- 
tical axis of the camera and passing a central point of 
the workpiece (an origin of a world coordinate system 
set to the workpiece) and rotation angles along the ro- 
tation axis are set. Since the optical axis of the camera 
is set parallel to the Z axis of the world coordinate sys- 
tem, an axis parallel to either the X-axis or the Y-axis of 
the world coordinate system, which is perpendicular to 
the Z axis, can be selected and the workpiece is rotated 



around the rotation axis at the workpiece position. 
[0034] In the example, an axis parallel to the X-axis 
of the world coordinate system is set as the rotation axis, 
and for the position/posture shown in FIG. 2b, the rota- 
tion angle of 30° is set to rotate the camera by 30° with 
respect to the workpiece along the rotation axis. A first 
reference model is created based on the image data of 
the workpiece at the position/orientation (posture) 
shown in FIG. 2b. Similarly, as shown in FIGS. 2c and 
2d: the camera 21 is rotated by 60° and 90°, respective- 
ly, along the rotation axis for capturing images of the 
workpiece to create second and third reference models. 
Only posture (orientation) information may be stored to 
be associated with each reference model in the case 
where the relative position between the camera and the 
reference workpiece is not changed, as in this example. 
[0035] Further, in this example, rotation angles of zero 
degree, 30 degrees, 60 degrees and 90 degrees are set 
for creating four reference models. The dividing range 
of the rotation angles may be set more finely and/or 
range of the rotation angle may be set greater to create 
more reference models for more precise detection of the 
position/posture of the workpiece. 
[0036] Processing for creating the four reference 
models will be explained referring to flowchart of FIG. 5. 
[0037] As described above, the 0-th position/posture 
of the robot at which the camera 20 captures the image 
of the object, and the rotation axis and the rotation an- 
gles with respect to the 0-th position/posture are set in 
advance in order to determine the subsequent positions/ 
postures of the reference workpiece, and also the 
number of the subsequent positions/postures of the 
workpiece are set. For intelligible explanation, it is as- 
sumed that the optical axis of the camera is parallel to 
the Y-axis of the world coordinate system and that a po- 
sition where the X-axis and Y-axis coordinate values are 
identical to those of the reference workpiece arranged 
at a predetermined position with a predetermined pos- 
ture and only the Z-axis coordinate value is different 
form that of the position of the reference workpiece 
taught to the robot as the 0-th image capturing position 
for obtaining the 0-th reference model. Further, the po- 
sitions of the robot where the camera is rotated with re- 
spect to the reference workpiece by 30 degrees, 60 de- 
grees and 90 degrees along the axis passing the central 
point of the reference workpiece and parallel to the X- 
axis of the world coordinate system are set as the 1st, 
2nd and 3rd image capturing positions, and the number 
N of the image capturing positions is set "4." 
[0038] When a command for creating reference mod- 
els is inputted from teaching operation panel 4, the proc- 
essor 1 of the robot controller 10 sets a counter M for 
counting the number of the image capturing to "0" (Step 
100). The robot is operated to have the M-th positionl- 
posture and a command for image capturing is output- 
ted to the image processing apparatus 30 (Step 101). 
In response to this command, the image processing ap- 
paratus 30 performs capturing of an image of the refer- 
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ence workpisce with the camera 20 and the captured 
image data is stored in the frame memory 37. Further, 
relative position/orientation (posture) of the workpiece 
with respect to the camera is obtained and stored in the 
nonvolatile memory 37 as relative position/orientation 
(posture) of M-th reference model (Step 103). This rel- 
ative position/posture is hereinafter referred to as "first- 
sensor relative position/posture". Thus, position/pos- 
ture of the workpiece in a camera coordinate system set 
to the camera is obtained from the position/posture of 
the camera and the position/posture of the reference 
workpiece in the world coordinate system when captur- 
ing the image by the camera, and is stored as the rela- 
tive position/posture of the workpiece with respect to the 
camera (the first-sensor relative position/posture). For 
example, the position/posture of the workpiece in the 
camera coordinate system is stored as [xO, yO, zO, c/.0, 
PO, t0]c : where a, (3 and y mean rotation angle around 
X-, Y-, Z- axes, and "c" means the camera coordinate 
system. Further, as described above, in the case where 
the position of the camera relative to the workpiece is 
not changed, only the orientation(posture) of the camera 
[xO, yO, zO, a0, po, ~,0]c is stored to be associated with 
each reference model as the relative position/posture of 
the first sensor. 

[0039] The relative position/posture of (the tool center 
point of) the robot to be situated with respect to the work- 
piece, or the relative position/posture of the second vis- 
ual sensor 22 to be situated with respect to the work- 
piece is stored to be associated with M-th reference 
model, for a subsequent operation performed by the ro- 
bot, i.e., a precise detection of position/posture of the 
workpiece by the second visual sensor 22. This relative 
position/posture is hereinafter referred to as "work- 
piece-robot (second sensor) relative position/posture". 
Then, a data-captured signal is sent to the robot con- 
troller 10 (Step 104). 

10040] The workpiece-robot (second sensor) relative 
position/posture is for determinin g orientation and/or po- 
sition of a subsequent operation of the robbt RB, i.e., 
the orientation/position of the robot at which the three- 
dimensional sensor 22 can detect the object or a char- 
acteristic portion of the object, more precisely. Alterna- 
tively, in the case where the precise detection of the 
workpiece by the three-dimensional sensor 22 is omit- 
ted, the workpiece-robot relative position/posture may 
be set to the relative position/posture between the tool 
center point and the workpiece which is most suitable 
for picking up the workpiece by a robot hand. 
[0041] The workpiece-robot (second sensor) relative 
position/posture is stored as position/posture of the tool 
center point or the three-dimensional visual sensor 22 
in a workpiece coordinate system set to the workpiece. 
The following explanation is given assuming that the rel- 
ative position/posture of the three-dimensional visual 
sensor 22- is stored. The workpiece-robot (second sen- 
sor) relative position/posture is represented by an ap- 
proach vector for determining approach position/pos- 



ture of the three-dimensional visual sensor 22 with re- 
spect to the workpiece in precise measuring position/ 
posture of the workpiece W by the three-dimensional 
visual sensor 22. One approach vector may be com- 
5 monly set to reference models created for one kind of 
reference workpiece, since the optimal position/posture 
of the tool center point with respect to the workpiece can 
be determined unequivocally in accordance with a kind 
of the workpiece for precisely measuring the position/ 
10 posture of the workpiece by the three-dimensional vis- 
ual sensor 22 or picking up the workpiece by the robot 
hand. The approach vector may be selected from a plu- 
rality of patterns prepared in advance and may include 
orientation (posture) information only. 
*s [0042] The workpiece-robot (second sensor) relative 
position/posture can be expressed by three compo- 
nents (X, Y, 2)wp for parallel motion and three compo- 
nents (a, p, y)wp for rotation, where wp means the work- 
piece coordinate system. In this embodiment, the work- 
20 piece-robot (second sensor) relative posilion/poslure is 
commonly set for all the reference models. Further The 
information to be stored in Steps 103 and 104 may be 
gathered as the relative position/posture of the second 
visual sensor to be situated with respect to the CCD 
- 5 camera 21 . 

[0043] Upon receipt of the data-captured signal, the 
processor 1 of the robot controller 10 incrementally in- 
creases the value of the counter M by u 1 " (Step 105) and 
determines whether or not the value of the counter M is 
30 less than a set value N (=4) (Step 106). If the value of 
the counter M is less than the set value N, the procedure 
returns to Step 101 to move the robot to the M-th image- 
capturing position/posture. Processing of Steps 101 and 
the subsequent Steps are repeatedly executed until the 
35 value of the counter M equals the set value N(=4). 

[0044] Thus, the reference models are stored in the 
nonvolatile memory 38 and also the first-sensor relative 
position/posture, which is the relative position/posture 
of the first visual sensor 21 and the workpiece W and 
•*o the workpiece-robot (second sensor) relative position/ 
posture, which is the position/posture of the second vis- 
ual sensor 22 (or the robot) to be situated with respect 
to the workpiece Ware stored in the nonvolatile memory 
38. 

-fs [0045] The reference models may be created from a 
part of the image data of the reference object, and may 
be created by processing the image data of the refer- 
ence object. 

[0046] FIG. 7 shows an example of the first-sensor 
50 relative position/posture and the workpiece-robot (sec- 
ond sensor) relative position/posture in the case of four 
reference models. 

[0047] The first-sensor relative position/posture is ex- 
pressed as position/posture of the workpiece W in the 
55 camera coordinate system (X, Y, Z, a, p, y)c set for the 
CCD camera 21 . The relative positions of the workpiece 
W. i.e., the positions of the 0-th to 3rd reference models 
are the same, being expressed as (X, Y, Z)=(10.50, 
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-20.80, 50.50) and. the rotation angles a for 0-th to 3rd 
reference models for the rotation around the X-axis be- 
ing set to 0, 60 and 90 degrees, respectively. The CCD 
camera is rotated around the axis parallel to the X-axis 
of the world coordinate system with a center of rotation s 
set to the origin of the workpiece coordinate system., and 
since the X-axis of the camera coordinate system is set 
parallel to the X-axis of the world coordinate system, on- 
ly the rotation angles a in the camera coordinate system 
are changed. 10 
[0048] The workpiece-robot (second sensor) relative 
position/posture is expressed as the position/post urc of 
the second visual sensor 22 in the workpiece coordinate 
system (X, Y, Z, a, p, y)wp set to the wcrkc iccc ana the 
approach vector (30.5, 20.5, 60.9, 0.0 ; 0 0 0 Ojwp is set 
to represent constant orientation with respect to the 
workpiece W. 

[0049] The reference models and the relative posi- 
tion/posture ot the workpiece W and the camera 20 aie 
stored in the nonvolatile memory 3£ ol the irnacje 30 
processing apparatus 30. In the above de^cnUed em- 
bodiment, the reference models are created using a ro- 
bot, although the reference models may be created b/ 
a manual operation without using a robot In this c;isc 
the reference workpiece is arranged within a hcldol view -5 
of the camera connected to the image processing ap- 
paratus 30, and the images of the workpiece with differ- 
ent postures are captured by the camera. The reference 
models are created based on the image data and iho 
relative positions/postures of the camera and the work- 30 
piece at the image capturing are manually inputted, and 
are stored with the respective relative positions/pos- 
tures. 

[0050] Hereinafter, a picking-up operation for taking 
out an individual workpiece by a robot from a pile of 35 
workpieces each having a shape identical to that of the 
reference workpiece will be described, as an example 
of a method of detecting three-dimensionpl position/ 
posture of an object, using the image processing appa- 
ratus 30 storing the reference models. ] -w 
[0051] FIG. 6 is a flowchart of processing for the pick- 
ing-up operation using the reference models. 
[0052] When a picking-up command is inputted into 
the robot controller 1 0 from the teaching operation panel 
4, the processor 1 operates the robot RB to move the 
camera attached to the robot wrist to an image capturing 
position where a pile of workpieces are within a field of 
view of the CCD camera 21 (Step 200). Three-dimen- 
sional position/posture of the camera 21 on the world 
coordinate system at this image capturing position is so 
outputted to the image processing apparatus 30, and an 
image capturing command is outputted (Step 201 ). Up- 
on receipt of the image capturing.command, the proc- 
essor 31 of the image processing apparatus 30 captures 
an image of the pile of the workpieces W to obtain image 55 
data of some workpieces W and store the data in the 
frame memory 37 (Step 202). 

[0053] Then, , pattern matching processing is per- 



formed for the image data stored in the frame memory 
37 using one of the reference models (first reference 
model) stored in the nonvolatile memory 37 so as to de- 
tect a workpiece W (Step 203). In this pattern matching 
processing, matching of the image data of the reference 
model with the image data of workpieces is performed 
on the basis of position, turn and scale. A determination 
I made as to whether or not an object has a matching 
value equal to or greater than the set value (Step 204). 
If an object having a matching value equal or greater 
than the set value is not detected, the procedure pro- 
ceeds to Step 205 to determine whether or not the pat- 
tern matching is performed using all the reference mod- 
els (first to fourth reference models). If the pattern 
matching using all the reference models has not yet 
been achieved, further pattern matching is performed 
using another reference model (Step 206). 
[0054] If it is determined in Step 204 that an object 
having a matching value equal or greater than the set 
value with respect to any of the reference models is de- 
tected, the procedure proceeds to Step 207 to perform 
matching processing on the two-dimensional data of the 
detected workpieces W. using every taught mode. In 
Step 208, the reference model having the largest match- 
ing value in the pattern matching processing is selected, 
and the relative position/postu re of the workpiece W with 
respect to the camera 21 is determined based on the 
first-sensor relative position/posture, i.e., the relative 
position/posture of the camera and the reference work- 
piece stored for the selected reference model, and po- 
sition, rotation angle and scale of the image of the work- 
piece in the matching processing as well as data of the 
workpiece-robot (second sensor) relative position/pos- 
ture* associated with the selected reference model, 
which represent the position/posture of the second sen- 
sor 22 to be situated with respect to the workpiece are 
read from the nonvolatile memory 37 (Step 208). 
[0055] The reference model having the highest 
matching value is selected in this embodiment, although 
a reference model of the rotation angle of zero degree 
(the O-th reference model) may be selected with prec- 
edence, or an object having the highest expansion rate 
of scale (the object which is nearest to the camera, i.e. 
located at the top of the pile in this example) may be 
selected with precedence. 

[0056] The position and posture (orientation) of the 
detected workpiece on the world coordinate system is 
determined from the position and posture of the camera 
21 in the world coordinate system, which has been sent 
in Step 201 , and the relative position/posture of the 
workpiece W with respect to the camera 21 , and is out- 
putted (Step 209). Thus, since the relative position/pos- 
ture of the workpiece W. with respect to the camera 21 
is the position/posture of the workpiece W in the camera 
coordinate system, the position and posture (orienta- 
tion) of the detected workpiece W in the world coordi- 
nate system is obtained by an arithmetical operation of 
coordinate transformation using the data of the position/ 
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posture of the workpiece W in the camera coordinate 
system and the position/posture of the camera 20 in the 
world coordinate system (Step 209). 
[0057] The position/posture of the second visual sen- 
sor 22 to be situated for a subsequent operation in the 
world coordinate system is determined based on the de- 
termined position/posture of the detected workpiece W 
in the world coordinate system and the workpiece -robot 
(second sensor) relative position/posture data (ap- 
proach vector) (Step 210). The processor 1 operates the 
robot to situate the second visual sensor 22 to have the 
determined posit ion/posture, and outputs a measuring 
command to the second visual sensor 22 (Step 211). 
[0058] Upon receipt of the measuring command, the 
three-dimensional sensor 22 measures a three-dimen- 
sional position/posture of the workpiece W. Since the 
second visual sensor 22 is situated at a suitable posi- 
tion/posture with respect to the workpiece W designated 
by the approach vector, the three-dimensional position/ 
posture of the workpiece W can be precisely measured. 
The processor 31 of the image processing apparatus 30 
outputs the result of measurement to the robot controller 
10 (Step 212). 

[0059] The robot controller 10 operates the robot to 
perform a picking-up operation to grip and hold the de- 
tected workpiece W and move the held workpiece W to 
a predetermined position, based on the result of meas- 
urement by the three-dimensional visual sensor 22 
(Step 213). Then, the procedure returns to Step 202 to 
repeatedly execute the processing of Step 202 and sub- 
sequent Steps. 

[0060] When all the workpieces have been picked up 
from the pile of the workpieces, a matching value equal 
to or greater than the set reference value cannot be ob- 
tained in the pattern matching processing for all refer- 
ence models in Steps 203-206, and the picking-up op- 
eration is terminated. 

[0061] In the case where a pile of the workpieces can- 
not fall within the field of view of the camera 20, or in the 
case where it is not necessary to capture an image of a 
workpiece behind other workpieces by changing the ori- 
entation of the camera, the procedure may return to 
Step 200 when it is determined "Yes" in Step 205, to 
move the camera to another position/posture at which 
an image of the objective workpiece can be captured. 
[0062] In the foregoing embodiment, the three-dimen- 
sional sensor 22 is adopted as the second visual sensor 
for precisely detecting the position/posture of the work- 
piece, although the two-dimensional sensor may be 
adopted instead of the three-dimensional sensor. In the 
case where the two-dimensional sensor is adopted as 
the second visual sensor, the second sensor may be 
provided in addition to the first visual sensor, or the first 
visual sensor may function as the second visual sensor. 
[0063] The position/posture of the individual work- 
piece in the pile of workpieces is roughly detected by 
the first visual sensor and the precise position/posture 
of the detected workpiece is detected by the second vis- 



ual sensor situated at the suitable position/posture 
which is nearer to the workpiece W s to improve the pre- 
cision of detection. Therefore, when the first visual sen- 
sor takes place of the second visual sensor precise po- 

s sition/posture of the workpiece can be detected by cap- 
turing the image of the detected workpiece Wfrom the 
shorter distance and performing the matching process- 
ing by the CCD camera as the second sensor. 
[0064] Further, in the case where a wide-angle lens is 

to installed in the CCD camera as the image capturing de- 
vice, for example there is possibility of judging the incli- 
nation angle to be 30 degrees by influence of parallax 
when a workpiece of zero degree inclination is at a cor- 
ner of a field of view of the camera. In such a case, the 

75 camera may be moved in parallel in accordance with the 
position of the workpiece in the field of view of the cam- 
era to a position right above the workpiece to lose the 
influence of parallax, and at this position the image cap- 
turing processing of Step 201 and the subsequent Steps 

20 in FIG. 6 is performed so lhal false judgment is prevent- 
ed. 

[0065] Furthermore, in the foregoing embodiment, the 
first visual sensor is mounted on the distal end of the 
robot wrist, although the first visual sensor may be a sta- 
25 tionary camera fixed at a place above the pile of the 
workpieces. In this arrangement, the distance between 
the camera and the workpieces is lengthened to prevent 
the influence of parallax. Also, it is not necessary to op- 
erate the robot to capture the image of the workpieces 
so by the first visual sensor to shorten the cycle time. 

[0066] In the foregoing embodiment, one kind of ob- 
ject (workpieces) are detected and picked up, although, 
the rpbot system can be modified to detect and pick up 
plural kinds of objects (workpieces). 
35 [0067] In this case, Ma is the number of reference 
models created from a reference workpiece of a kind A 
and Mb is the number of reference models created from 
a reference workpiece of a kind B and information of the 
workpiece kinds A or B are additionally stored to be as- 
^0 sociated with each reference model. Then, matching 
processing is carried out between the captured image 
and the (Ma+Mb) number of reference models to select 
the matched reference model, and the workpiece kind 
information, in addition to the orientation and/or position 
^5 for the subsequent operation, are determined. In this 
way, it is not necessary to separate the workpieces ac- 
cording to the kind of workpieces in advance, so that the 
mixed kinds of workpiece can be picked up one by one 
to reduce the operational cost. 
so [0068] In the foregoing embodiment, the position/pos- 
ture of the object is roughly determined based on image 
data captured by the first visual sensor and the precise 
position/posture of the detected workpiece is measured 
by the second visual sensor, although, the second im- 
55 age capturing device may be omitted and the orientation 
and/or position of the robot operation can be determined 
using the first image data capturing device only In this 
case, the relative position/posture of the robot (tool cent- 
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er point) to be situated with respect to the workpiece is 
stored as the workpiece-robot relative position/posture. 
[0069] According to the present invention, a position/ 
posture of an objective workpiece in a randomly ar- 
ranged pile of workpieces of one kind or a plurality of s 
kinds, or an aggregation of workpieces of one kind or a 
plurality of kinds gathered in a predetermined region, 
which have different three-dimensional positions/pos- 
tures, is-detected, and an orientation/position for an op- 
eration on the detected workpiece by a robot is deter- io 
mined. Further, the second image data capturing device 
measures the position/posture of the workpiece more 
precisely at the determined position/posture (approach 
vector) for the robot operation. Therefore, the robot can 
securely perform a picking-up operation of picking up is 
an individual workpiece from such a pile or an aggrega- 
tion. 



Claims 20 

1. A robot system having an image processing func- 
tion for determining posture, or posture and position 
of a robot operation on an object comprising: 

25 

a robot; 

a first image capturing device; 3. 
a memory storing reference model created ac- 
cording to image data of a reference object cap- 
tured by said image capturing device in a plu- 30 
rality of different directions, and storing infor- 
mation of the capturing directions to be respec- 4. 
tively associated with said reference models, 
and information of orientation of the robot op- 
eration with respect to the object, said refer- 35 
ence object being the object for detection or an 
object having a shape identical to that of the 5. 
object for detection; and j 
a processor to perform matching processing on 
image data containing an image of ! ? the object 40 
for detection captured by said image capturing 
device with said reference models to select an 6. 
image of an object matched with one of said 
reference models, and to determine orienta- 
tion, or orientation and position, of an operation 45 
to be performed by the robot based on the se- 
lected image of the object, said one reference 7. 
model and the information about the capturing 
direction and information about the orientation 
of the robot operation with respect to the object so 
associated with said one reference model. 

2. A robot system having an image processing func- 
tion for determining posture, or posture and position 

of a robot operation on an object comprising: ss 

a robot; 

a first image capturing device; 



a memory storing reference model created ac- 
cording to image data of different kinds of ref- 
erence objects captured by said first image 
capturing device, and storing information of the 
kinds respectively associated with said refer- 
ence models, and information about orientation 
of the robot operation with respect to the object 
of each kind, each of said reference objects be- 
ing an object of operation of one such kind or 
an object having a shape identical to that of the 
object of operation of one such kind; and 
a processor to perform matching processing on 
image data containing an image of the object 
for operation captured by said first image cap- 
turing device with said reference models to se- 
lect an image of an object matched with one of 
said reference models, and to determine orien- 
tation, or orientation and position of the robot 
operation based on the selected image of the 
object, said one reference model, the informa- 
tion about the kind of information associated 
with said one reference model and the informa- 
tion of the orientation about the robot operation 
respect to the object associated with said one 
reference model. 

A robot system having an image processing func- 
tion according to claim 1 or 2, wherein said refer- 
ence models are obtained from a part of the image 
data of the reference object. 

A robot system having an image processing func- 
tion according to claim 1 or 2, wherein said refer- 
ence models are obtained by processing the image 
data of the reference object. 

A robot system having an image processing func- 
tion according to any preceding claim, wherein said 
first image capturing device comprises a camera for 
capturing two-dimensional image data. 

A robot system having an image processing func- 
tion according to claim 5, wherein said image data 
of the reference object are captured by said camera .. 
from a predetermined distance, 

A robot system having an image processing func- 
tion according to any preceding claim, wherein said 
robot system further comprises a second image 
capturing device and 

said robot situates said second image data cap- 
turing device to have said determined orienta- 
tion or to have said determined orientation and 
said determined position with respect to the ob- 
ject, and 

said processor processes second image data 
captured by said second image capturing de- 
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vice to detect position and/or posture of the ob- 
ject with respect to said second image data 
capturing device 

8. A robot system having an image processing tunc- 5 

tion according to any one of claims 1 to 6 S wherein 

said robot system further comprises a second im- 

age capturing device for obtaining three-dimension- 
al position; 

10 

said robot situates said second image da ta cap- 
turing device to have said determined orienta- 
tion or to have said determined orientation nnd 
said determined position with respect to the ob- 
ject, so that said second image data capturing is 
device is directed to a characterising pert ion ol 
the object; and 

said processor detects three-dimcnsion.Hl posi- 
tion and/or posture of the object dhscc on 
three-dimensional position of said charHcloriS:- 30 
ing portion obtained by said second irrinyu enp- 
turing device. 

9. A robot system having an image processing tunc 

tion according to claim 7 or 8, wherein said hrci im 35 
age data capturing device is used as said second 
image data capturing device. 

10. A robot system having an image processing func- 
tion according to claim 7, 8 or 9, wherein said sec- 30 
ond image capturing device comprises a three-di- 
mensional visual sensor of spot-light scanning type 
capable of measuring distance between the sensor 
and an object. 

35 

11. A robot system having an image processing func- 
tion according to claim 7, 8 or 9, wherein said sec- 
ond image data capturing device comprpes a struc- 
tured-light unit for irradiating a structure^ light on an 
object and capturing an image of the object includ- 40 
ing the irradiated light on the object. 

12. A robot system having an image processing func- 
tion according to any preceding claim, wherein said, 
robot operation is an operation for picking up at 4S 
least one object from a plurality of objects over- 
lapped with each other. 
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